quasi-Newton Optimization Method Research Articles

Q&A With Dennis Denney, JPT Technology Focus Creator

_ In 1997, JPT debuted a monthly feature that represented an unconventional approach in meeting SPE’s goal of disseminating technical information to its members. Soon titled “Technology Focus,” the feature brought together subject-matter experts (SMEs) in a range of important industry topics to review technical-paper abstracts gathered from the previous year of SPE meetings, as well as the Offshore Technology Conference. Its architect, engineer, and pilot was Technology Editor Dennis Denney, who would become a familiar presence to SPE members and staff alike over a 17-year career, although his industry roots extended to his college days. Dennis helped recruit SMEs for review service; sent them hundreds of abstracts and full papers (monthly, that is; the cumulative count of papers he handled is likely a six-digit figure); answered their questions and concerns with prompt, supportive guidance; and then, once papers had been selected, synopsized them for publication in the magazine with the longtime help of Assistant Technology Editor Karen Bybee. As this writer well knows, condensing a 10,000-word paper on, say, distributed quasi-Newton derivative-free optimization methods for field development optimization into a 1,500-word summary that captures the novelty and technical sophistication of its authors’ work can be a daunting task. Applying his own background as a petroleum engineer to methodically analyze and trim each chosen paper, Dennis did just that dozens of times a year, transforming these texts into svelte packages that included only one table or figure to deliver the highest possible quantity of quality to JPT readers. When Dennis retired in 2013, I initially split his eyebrow-raising workload with fellow Technology Editor Adam Wilson (now JPT’s Special Publications Editor). It was immediately clear from the way that reviewers spoke about Dennis, and his contributions to the magazine and SPE itself, that he was held in the same high regard outside of the organization as he was within it, a testament to the reputation he had earned while working with so many members and SMEs. It was entirely fitting that, for JPT’s 75th anniversary celebration, we caught up with Dennis to reflect on his achievements in the industry and with the groundbreaking Technology Focus feature he created. Dennis lives with his wife Linda in Rockwall, Texas—not too far from SPE headquarters in Dallas, but not too close, either!—enjoying his retirement and his family. (Note: For more on the development and history of the Technology Focus features, read the companion piece to this article, “Technology Focus Topics Reflect Industry Growth, Evolution Over 25+ Years.”) Why did you decide to go into the field of petroleum engineering? I worked several years as an engineering tech, testing gas wells, tracking well and reservoir pressures, and writing computer programs to analyze pressure data. The department manager encouraged me to finish my degree and become a petroleum engineer. So, after I received a scholarship from J. Harold Dunn (at age 28), my wife and I sold our house, packed our things into a U-Haul truck, and loaded two young kids into the car to move to Bryan/College Station to attend Texas A&M University.

A single timescale stochastic quasi-Newton method for stochastic optimization

In this paper, we propose a single timescale stochastic quasi-Newton method for solving the stochastic optimization problems. The objective function of the problem is a composition of two smooth functions and their derivatives are not available. The algorithm sets to approximate sequences to estimate the gradient of the composite objective function and the inner function. The matrix correction parameters are given in BFGS update form for avoiding the assumption that Hessian matrix of objective is positive definite. We show the global convergence of the algorithm. The algorithm achieves the complexity to find an approximate stationary point and ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance ϵ. The numerical results of nonconvex binary classification problem using the support vector machine and a multicall classification problem using neural networks are reported to show the effectiveness of the algorithm.

Quasi-Newton methods for topology optimization using a level-set method

The ability to efficiently solve topology optimization problems is of great importance for many practical applications. Hence, there is a demand for efficient solution algorithms. In this paper, we propose novel quasi-Newton methods for solving PDE-constrained topology optimization problems. Our approach is based on and extends the popular solution algorithm of Amstutz and Andrä (J Comput Phys 216: 573–588, 2006). To do so, we introduce a new perspective on the commonly used evolution equation for the level-set method, which allows us to derive our quasi-Newton methods for topology optimization. We investigate the performance of the proposed methods numerically for the following examples: Inverse topology optimization problems constrained by linear and semilinear elliptic Poisson problems, compliance minimization in linear elasticity, and the optimization of fluids in Navier–Stokes flow, where we compare them to current state-of-the-art methods. Our results show that the proposed solution algorithms significantly outperform the other considered methods: They require substantially less iterations to find a optimizer while demanding only slightly more resources per iteration. This shows that our proposed methods are highly attractive solution methods in the field of topology optimization.

Multi-pattern synthesis in fourth-dimensional antenna arrays using BGM-based quasi-Newton memetic optimization method

Abstract The advancement in wireless communication is fueling the growth of innovative antenna array designs toward cost-effective and performance-oriented solutions. This paper proposed unconventional methods to design antenna arrays for multi-pattern synthesis without using attenuators or phase shifters. A low-cost alternative is proposed with “Time-modulation”-based antenna array capable of electronic scanning and beam steering. Here, “Time” is utilized as a fourth-dimensional (4D) array parameter, and that is why “Time-modulated” arrays are also called as 4D antenna arrays. The idea is to control the high-speed switch attached with each antenna periodically to produce desirable current and phase tapering. This article expanded the “Time-modulation” concept to synthesize multiple radiation patterns like monopulse patterns, scanned beam patterns, shaped beam patterns, and cosecant-squared beam patterns for multifunctional radar systems. Suitable time schemes are developed to generate the narrowband sum–difference patterns useful for monopulse radars. Simultaneous scanned beam patterns are also proposed for narrowband communication. Furthermore, to address the wideband applications, shaped flat-top beam patterns and cosecant-squared beam patterns are also proposed. In this regard, 20- and 16-element “Time-modulated” linear array antennas are developed, and the parameters of the arrays are controlled by suitably designed objective functions with quasi-Newton method (QNM)-based memetic optimization method. For this purpose, first a well-known genetic algorithm is adopted to search the potential trust regions in the exploration stage and QNM is used for fine-tuning. Furthermore, the Broyden’s good method-based direction-updating equation is used with QNM to improve the performance.

Determination of a Nonlinear Coefficient in a Time-Fractional Diffusion Equation

In this paper, we study direct and inverse problems for a nonlinear time fractional diffusion equation. We prove that the direct problem has a unique weak solution and the solution depends continuously on the coefficient. Then we show that the inverse problem has a quasi-solution. The direct problem is solved by the method of lines using an operator approach. A quasi-Newton optimization method is used for the numerical solution to the inverse problem. The Tikhonov regularization is used to overcome the ill-posedness of the inverse problem. Numerical examples with noise-free and noisy data illustrate the applicability and accuracy of the proposed method to some extent.

Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization

Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization

A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions

In this paper, we propose a natural gradient descent algorithm with momentum based on Dirichlet distributions to speed up the training of neural networks. This approach takes into account not only the direction of the gradients, but also the convexity of the minimized function, which significantly accelerates the process of searching for the extremes. Calculations of natural gradients based on Dirichlet distributions are presented, with the proposed approach introduced into an error backpropagation scheme. The results of image recognition and time series forecasting during the experiments show that the proposed approach gives higher accuracy and does not require a large number of iterations to minimize loss functions compared to the methods of stochastic gradient descent, adaptive moment estimation and adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization.

Passive periodic motion of an asymmetric spring loaded inverted pendulum hopping robot

For improving the energy efficiency of hopping robot, an asymmetric spring loaded inverted pendulum hopping model with leg mass is considered. The period orbit problem of two-legged hopping robot is investigated. Firstly, the hybrid dynamic model is constructed. Then the passive hopping gaits are found using quasi-newton optimization method. Secondly, a PD controller is implemented to track the desired pitch trajectory of the body. Through applying control during stance phase, period orbits of the robot with offset body mass is obtained. Finally, the effect of the location of the leg mass and the body mass on hopping performances is investigated.

On the implementation of a quasi-Newton interior-point method for PDE-constrained optimization using finite element discretizations

ABSTRACT We present a quasi-Newton interior-point method appropriate for optimization problems with pointwise inequality constraints in Hilbert function spaces. Among others, our methodology applies to optimization problems constrained by partial differential equations (PDEs) that are posed in a reduced-space formulation and have bounds or inequality constraints on the optimized parameter function. We first introduce the formalization of an infinite-dimensional quasi-Newton interior-point algorithm using secant BFGS updates and then proceed to derive a discretized interior-point method capable of working with a wide range of finite element discretization schemes. We also discuss and address mathematical and software interface issues that are pervasive when existing off-the-shelf PDE solvers are to be used with off-the-shelf nonlinear programming solvers. Finally, we elaborate on the numerical and parallel computing strengths and limitations of the proposed methodology on several classes of PDE-constrained problems.

Extrapolating Long-Run Yield Curves: An Innovative and Consistent Approach

This article proposes a method to build term structures that are consistent with market data and that provide interest rates for which the volatility, on average, decreases as maturities increase. The method is designed for continuous repetitive use and is consistent with work by Diebold and Li, providing reasonable extrapolated rates, with an appropriate level of volatility over time. The Svensson model is adopted, and its parameters are estimated by the combination of a genetic algorithm and a quasi-Newton nonlinear optimization method. We innovate with a new objective function that focuses on both parts of the estimated curves (interpolated and extrapolated). For this purpose, a stability component is added. The new objective function aims to solve the problem of estimating long-term rates not observable in the market, for which the estimates are usually artificially stable or excessively volatile. The results show that the estimation method is able to bring the volatility of extrapolated rates to levels consistent with those observed for the longest liquid rate. Estimation errors are small enough and there is no statistical evidence that they are biased. The method is useful for the insurance market, since it provides interest rates that do not lead to artificially stable or excessively volatile technical provisions.

A modified secant equation quasi-Newton method for unconstrained optimization

A modified secant equation quasi-Newton method for unconstrained optimization

Multi-Objective Optimal Design and Development of a Four-Bar Mechanism for Weed Control

Weeds compete with crops for water, nutrients, and light consequently, have adverse effects on the crop yield and overall productivity. Mechanical weeding is the most common non-chemical method for weed control, which is applied in organic farming, and the weed cultivator is the most common implement in mechanical weeding. This study aimed to design and develop an innovative active tool to optimize the cultivation depth, which can avoid damage to crop roots and improve the key performance indicators of an inter-row cultivator. A quasi-Newton optimization method and a hybrid of the non-dominated sorting genetic algorithm (NSGA-II) and goal attainment method were separately applied to synthesize and develop a four-bar mechanism for weeding requirements. The transmission angle of the mechanism and the desired path of the weeding blade were simultaneously optimized using these multi-objective optimization techniques. The performance of the developed four-bar cultivator based on the optimization techniques was compared with the ones developed based on the classic methods and also with several conventional tools evaluated in other studies. The results showed that applying the quasi-Newton optimization method and hybrid genetic algorithm can propose a more effective weed cultivator in terms of performance indicators, namely weeding performance, mechanical damage to crop plants and cultivation depth. In addition, the optimization of the transmission angle guaranteed the smooth rotations in the mechanism’s joints.

A Limited Memory BFGS Based Unimodular Sequence Design Algorithm for Spectrum-Aware Sensing Systems

Unimodular sequences with good correlation and spectral properties are desirable in numerous applications such as active remote sensing and communication systems. Therefore, designing sequences with stopband and correlation sidelobe constraints has gained a lot of attention in the last few decades. In this paper, we propose a fast and efficient iterative algorithm to design unimodular and sparse frequency waveforms with low aperiodic/periodic autocorrelation sidelobes and desired stopband properties. In our approach, the bi-objective optimization problem which minimizes both the integrated sidelobe level (ISL) of the autocorrelation function and the power density in the spectral stopbands is first turned into an unconstrained single objective optimization problem and then is treated as a nonlinear large-scale problem. For the solution of the problem, we develop an algorithm based on Limited Memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) Quasi-Newton optimization method. Unlike most gradient based algorithms which employ line searches to deduce the step length, owing to L-BFGS method, unit step length is taken as a general rule to avoid the cost of computation at every iteration with very few exceptions. The calculation of gradient is based on Fast Fourier Transform and Hadamard product operations and thus the algorithm is fast and computationally efficient. Moreover, the algorithm is space efficient and its low-memory feature makes it possible to generate long sequences. Several numerical examples are presented to validate the efficacy of the proposed method and to show its superiority over other state-of-art algorithms.

Distributed quasi-Newton derivative-free optimization method for optimization problems with multiple local optima

Distributed quasi-Newton derivative-free optimization method for optimization problems with multiple local optima

Optimization Approaches for Nonlinear State Space Models

The Local Model State Space Network (LMSSN) is a recently developed black box algorithm in nonlinear system identification. It has proven to be an appropriate tool on benchmark problems as well as for real-world processes. A severe shortcoming though is the long computation time that is necessary for model training. Therefore, a different optimization strategy, the adaptive moment estimation (ADAM) method with mini batches is used for the LMSSN and compared to the current Quasi-Newton (QN) optimization method. It is shown on a numerical Hammerstein example and on a well known Wiener-Hammerstein benchmark that the use of ADAM and mini batches does not limit the performance of the LMSSN algorithm and speeds up the nonlinear optimization per investigated split by more than 30 times. The price to be paid, however, is higher parameter variance (less interpretability) and more tedious hyperparameter tuning.

Navigator function for the conformal bootstrap

Current numerical conformal bootstrap techniques carve out islands in theory space by repeatedly checking whether points are allowed or excluded. We propose a new method for searching theory space that replaces the binary information "allowed"/"excluded" with a continuous "navigator" function that is negative in the allowed region and positive in the excluded region. Such a navigator function allows one to efficiently explore high-dimensional parameter spaces and smoothly sail towards any islands they may contain. The specific functions we introduce have several attractive features: they are well-defined in large regions of parameter space, can be computed with standard methods, and evaluation of their gradient is immediate due to an SDP gradient formula that we provide. The latter property allows for the use of efficient quasi-Newton optimization methods, which we illustrate by navigating towards the 3d Ising island.

Adaptive scaling damped BFGS method without gradient Lipschitz continuity

The Broyden–Fletcher–Goldfarb–Shanno (BFGS) method plays an important role among the quasi-Newton algorithms for nonconvex and unconstrained optimization problems. However, in the proof of global convergence, BFGS-type methods generally need to assume that the gradient of the objective function is Lipschitz continuous. This issue prompts us to try to find quasi-Newton method for gradient non-Lipschitz continuous and nonconvex optimization based on the classical BFGS formula. In this paper, we propose an adaptive scaling damped BFGS method for gradient non-Lipschitz continuous and nonconvex problems. With Armijo or Weak Wolfe–Powell (WWP) line search, global convergence can be obtained. Under suitable conditions the convergence rate is superlinear with WWP-type line search. Applications of the given algorithms include the tested optimization problems, which turn out the proposed method is powerful and promising.

A Comparison of Survivor Rate Estimates for Some Probability Distribution Models Using Least-Squares Method in Conjunction with Simplex and Quasi-Newton Optimization Methods

A Comparison of Survivor Rate Estimates for Some Probability Distribution Models Using Least-Squares Method in Conjunction with Simplex and Quasi-Newton Optimization Methods

A new modification of the quasi-newton method for unconstrained optimization

<p class="MsoNormal" style="text-align: justify;"><span>In this work we propose and analyze a hybrid conjugate gradient (CG) method in which the parameter <!--[if gte mso 9]><xml>
 <o:OLEObject Type="Embed" ProgID="Equation.3" ShapeID="_x0000_i1025"
 DrawAspect="Content" ObjectID="_1674222415">
 </o:OLEObject>
 </xml><![endif]-->is computed as a linear combination between Hager-Zhang [HZ] and Dai-Liao [DL] parameters. We use this proposed method to modify BFGS method and to prove the positive definiteness and QN-conditions of the matrix. Theoretical trils confirm that the new search directions aredescent directions under some conditions, as well as, the new search directions areglobally convergent using strong Wolfe conditions. The numerical experiments show that the proposed method is promising and outperforms alternative similar CG-methods using Dolan-Mor'e performance profile. </span><br /><br /></p>

Adjoint based state estimation of compressible flow in porous media

Abstract In this paper, we study the state estimation of compressible single phase flow in compressible porous media. The initial pressure distribution is estimated according to discrete adjoint approach based on the collected well pressure data. The first-order Tykhonov regularization method is used to obtain reasonable estimation. By analyzing the optimality condition of estimation problem, the discrete adjoint state equation and discrete adjoint gradient are derived based on the numerical scheme of the continuous equations. A quasi-Newton numerical optimization method related to adjoint gradient is proposed to solve the estimation problem. The estimation results with different regularization coefficients are compared and analyzed by numerical experiments. The deviation between the estimated pressure obtained without regularization and the real pressure is large. Estimation result with smaller deviation and higher smoothness can be obtained through appropriate regularization coefficient. When the observation error is large, the observed values generated by the estimated pressure fit well with the real pressure.